NVIDIA Performance Tools Explorer

What it is: A system-wide performance analysis tool that visualizes an application's algorithm and interactions between CPUs, GPUs, OS, and APIs on a unified timeline.

Key Use Cases:

• Identifying if an application is CPU-bound, I/O-bound, or lacks parallelism.
• Understanding CPU-GPU interactions and data transfers.
• Initial assessment of network communication and multi-node performance.
• Correlating activities across CPUs, GPUs, NICs, and OS.

Core Benefit: Serves as a "first-look" diagnostic to pinpoint the largest optimization opportunities by exposing system-level bottlenecks with low overhead.

Nsight Systems guides developers toward the true sources of inefficiency, preventing premature optimization of incorrect components.

Example Command(s):

# Basic profiling of an application
nsys profile ./my_application arg1 arg2

# Profile with specific traces (CUDA, NVTX) and statistics enabled
nsys profile -t cuda,nvtx --stats=true -o my_report ./my_application

What it is: An interactive kernel profiler for detailed analysis of CUDA and NVIDIA OptiX kernels.

Key Use Cases:

• Optimizing specific CUDA kernels by examining GPU throughput, warp state, instruction execution, and memory access patterns.
• API debugging and correlating performance data to source code (CUDA C/C++, PTX, SASS).
• Receiving guided analysis and actionable recommendations based on NVIDIA best practices.

Core Benefit: Provides indispensable granular detail to maximize individual GPU kernel efficiency, crucial after Nsight Systems identifies a GPU-bound workload.

Nsight Compute helps resolve performance limiters like instruction latency, memory bandwidth constraints, or low SM occupancy.

Example Command(s):

# Profile all kernels in an application and save to a report file
ncu -o my_kernel_report ./my_application

# Profile a specific kernel by name (regex supported)
ncu --kernels my_specific_kernel_name_regex ./my_application

# Collect a specific set of metrics/sections (e.g., MemoryWorkloadAnalysis)
ncu --set full --section MemoryWorkloadAnalysis ./my_application

Focus: Hardware fault detection and stability, particularly for server/data center GPUs.

Key Use Cases: Runs various tests on GPU memory to identify problematic memory modules or hardware faults.

Core Benefit: Helps ensure memory hardware integrity. Higher diagnostic levels provide more thorough testing.

Example Command(s):

# Run Level 2 diagnostics (includes memory tests) on all GPUs in group 1
sudo dcgmi diag -g 1 -r 2

# Run Level 3 diagnostics (more intensive memory tests)
sudo dcgmi diag -g 1 -r 3

Focus: Reporting ECC (Error Correcting Code) memory errors for GPUs that support this feature.

Key Use Cases: Checking for single-bit (corrected) and double-bit (uncorrected) ECC errors. Monitoring retired memory pages due to persistent errors.

Core Benefit: Provides visibility into the health and error status of ECC-enabled GPU memory.

Example Command(s):

# Query memory details, ECC status, and page retirement info
nvidia-smi -q -d MEMORY,ECC,PAGE_RETIREMENT

# Continuously monitor ECC errors (refreshes every second)
watch -n 1 "nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total --format=csv,noheader"

Focus: Debugging memory access errors within CUDA applications at the code level.

Key Use Cases: Detecting out-of-bounds global/shared memory accesses, misaligned accesses, uninitialized memory usage, and other memory-related bugs in CUDA kernels.

Core Benefit: Helps CUDA developers write more robust and correct code by pinpointing memory safety issues.

Example Command(s):

# Run memcheck tool on a CUDA application
compute-sanitizer --tool memcheck ./my_cuda_application

# Generate a report file
compute-sanitizer --tool memcheck --log-file memcheck_report.log ./my_cuda_application

Focus: Analyzing memory access patterns and performance within CUDA kernels, not for error detection but for optimization.

Key Use Cases: Identifying memory throughput bottlenecks, high-latency memory operations, inefficient cache utilization, and understanding how kernels interact with different levels of GPU memory.

Core Benefit: Guides optimization of kernel memory access to improve performance.

Example Command(s):

# Collect detailed memory workload analysis sections
ncu --set full --section MemoryWorkloadAnalysis --section SpeedOfLight_HierarchicalDoublePrecision ./my_cuda_kernel_app

# Focus on L1/L2 cache metrics
ncu --metrics l1tex__t_sectors_pipe_lsu_mem_global_op_ld_lookup_hit.sum,l1tex__t_sectors_pipe_lsu_mem_global_op_ld_lookup_miss.sum ./app

Focus: Visualizing and timing memory copy operations (e.g., `cudaMemcpy`, OpenACC data transfers) on the system timeline.

Key Use Cases: Identifying large or frequent CPU-GPU transfers, assessing if transfers overlap with computation, pinpointing PCIe bandwidth saturation, and analyzing the impact of memory transfer latency.

Core Benefit: Provides a clear view of data movement between host and device, highlighting inefficiencies that can be addressed by techniques like using pinned memory, asynchronous transfers, or reducing data volume.

Example Command(s):

# Profile CUDA memory operations along with other system activity
nsys profile -t cuda,nvtx,osrt -o transfer_analysis_report ./my_app_with_transfers

# Focus on CUDA API calls, including memory transfers
nsys profile --trace=cuda ./my_app_with_transfers

Focus: Benchmarking the peak achievable bandwidth of the PCIe bus for host-to-device (HtoD) and device-to-host (DtoH) transfers.

Key Use Cases: Verifying that the PCIe hardware configuration (e.g., link width, speed) allows for expected transfer rates. Identifying if the system is underperforming in terms of raw transfer capability.

Core Benefit: Establishes a baseline for maximum PCIe throughput, helping to distinguish between hardware limitations and application-level transfer inefficiencies.

Example Command(s) (using CUDA Sample `bandwidthTest`):

# (Typically found in CUDA Samples: /usr/local/cuda/samples/1_Utilities/bandwidthTest)
# Test all memory types and transfer directions
./bandwidthTest

# Test host-to-device bandwidth using pinned memory
./bandwidthTest --memory=pinned --mode=htod

# Test device-to-host bandwidth using pageable memory
./bandwidthTest --memory=pageable --mode=dtoh

Focus: Monitoring PCIe bandwidth utilization (bytes transmitted/received over PCIe) and link error counts at a system/cluster level.

Key Use Cases: Spotting GPUs with consistently saturated PCIe links or links accumulating errors, which could impact CPU-GPU transfer performance.

Core Benefit: Provides ongoing monitoring of PCIe health and throughput, complementing Nsight Systems' application-specific trace.

Example Command(s):

# Monitor PCIe transmit and receive bytes for GPU 0 in group 1
# Field IDs: 1006 (PCIe Tx Bytes), 1007 (PCIe Rx Bytes)
dcgmi dmon -g 1 -i 0 -e 1006,1007

# Get PCIe link generation and width for all GPUs
dcgmi discovery -c # Lists capabilities including PCIe gen and width

Focus: A suite of technologies enabling data transfers between GPUs and other devices (NICs, storage) to bypass the CPU and system memory, reducing latency and CPU overhead.

Key Use Cases: GPUDirect RDMA for network transfers, GPUDirect Storage for direct access to NVMe drives. While not a "tool" itself, applications and libraries utilize these technologies.

Core Benefit: Significantly improves performance for I/O intensive workloads by creating more direct data paths to/from the GPU.

Nsight Systems can help observe the benefits of GPUDirect (e.g., lower CPU usage during transfers, faster I/O operations) when it's effectively employed by the application or its libraries.